Analyzing NAEP/TIMSS Data with Direct Estimation using the R packages EdSurvey and Dire

<hr>

Analyzing NAEP/TIMSS Data with Direct Estimation using the R
========================================================

**Presenters**: Paul Bailey, Ting Zhang

_July 2022_

.logo[
<img src="data:image/png;base64,#presentationFigures/airLogo.png" width="80%" height="80%" />
]

---

Workshop Goal
========================================================
<br>
## Provide participants with an overview of the methods used to analyze national and international large-scale assessment data using the R package `EdSurvey` and `Dire`.

---

Outline of EdSurvey Workshop
========================================================

1. Introduction to R, EdSurvey, and Dire

2. Data Processing and Data Manipulation

3. Hands-on practice
  
4. Descriptive statistics

5. Hands-on practice

6. Direct estimation with EdSurvey and Dire

7. Hands-on practice

---

Course Materials
========================================================

Available here: [2022 IMPS Training Content](https://github.com/American-Institutes-for-Research/2022-IMPS-EdSurvey-Training/)

---
class: inverse, section-slide

Introduction to R, EdSurvey, and Dire
========================================================
<hr class="sectionSlide">
.logoSection[
<img src="data:image/png;base64,#presentationFigures/naepLogo.png" width="80%" height="80%" />
]

---

Why R?
========================================================

1. **Free:** users can legally use and edit R package code

2. **Extensible:** large variety of contributed packages that expand its functionality

3. **Reproducible:** automated data analysis

4. **Designed by and for researchers:** robust ecosystem to translate data into analyses, visualizations, and summary reports with one software

---

Why EdSurvey?
========================================================

1. **One-stop shop** for data downloading, processing, manipulation and analysis of survey data.

2. **Automated**: Weights and complex sampling design calculations are automated following standard OECD methodology.

3. **Simple**: e.g., a regression with 80 replicate weights requires only a few lines of code.

4. **Flexible**: You can use functions that rely on EdSurvey methods or get the data and use traditional R.

5. **Minimizes memory footprint** by only reading in required data.

---

Why Dire?
========================================================

1. **Score**: Assessments with the matrix booklet design require special considerations in data analysis, e.g., IRT and multiple imputation for item responses. `Dire` provides direct estimation functions that handles analyses of these assessment data properly.

2. **Efficient**: Students’ latent proficiency distribution, as well as reporting group difference parameters, are estimated on the fly.

3. **Plausible Values Generation**: No need to rely on testing companies. Plausible values can be generated from the user-defined MML model and used for further analysis.

4. **Expanding Research Scope**: Providing the opportunity for researchers to link administrative data, aggregate data about a community from official statistics, or data from multiple surveys to open up new research questions.

---

Basic R Infrastructure
========================================================

---

Basic R Infrastructure
========================================================

---

Basic R Infrastructure
========================================================

---

Basic R Infrastructure
========================================================

---
class: inverse, section-slide

Get to Know the R Environment
========================================================
<hr class="sectionSlide">
.logoSection[
<img src="data:image/png;base64,#presentationFigures/naepLogo.png" width="80%" height="80%" />
]

---

Follow Along - R Scripts
========================================================

![](data:image/png;base64,#presentationFigures/rstudioScript.png)

Follow along in [edsurvey_part1_Script.R](https://github.com/American-Institutes-for-Research/2022-IMPS-EdSurvey-Training/blob/main/edsurvey_part1_Script.R)

---

Notes About Using R
========================================================

- Highlight , `ctrl` + `enter` executes code to console

- Comment character is a hash

```r
# this line is not executed
```

- Variables are assigned with an equals or `<-`

```r
x <- 12
x
```

```
## [1] 12
```

- In file names on Windows use a forward slash
  - `C:/`
  
- R is case sensitive!

```r
j <- 12
J
```

```
## Error in eval(expr, envir, enclos): object 'J' not found
```

---

Notes About Using R (cont)
========================================================

- Any command can be input with a question mark preceding it to open the help guide

```r
?mean
```

- Use the up arrow on your keyboard to copy your previous lines of code

- Try tab completion, type, "data" then hit the tab key

![](data:image/png;base64,#presentationFigures/dataTab.png)

---

Using R Functions
========================================================

- `c()` function combines values, separated by a comma, into a vector

```r
colors <- c("red", "green", "blue")
colors
```

```
## [1] "red"   "green" "blue"
```

```r
numbers <- c(1, 2, 3)
numbers
```

```
## [1] 1 2 3
```

- In the `EdSurvey` package we'll use vectors to combine the names of variables in our analyses

---

Using R Functions
========================================================

- Arguments can be explicitly or implicitly named

```r
mean(x = numbers)
```

```
## [1] 2
```

```r
mean(numbers)
```

```
## [1] 2
```

- Arguments are separated by commas

---

Installing the EdSurvey Package
========================================================

- After opening up RStudio, run the following scripts in the console to download and initialize the `EdSurvey` package:

```r
#install Dire 2.0.1
# you may need to get rtools
install.packages("Dire")

# then install devtools and EdSurvey from GitHub
install.packages("devtools")
devtools::install_github("American-Institutes-for-Research/edsurvey")
```

```r
# to load the package
library(EdSurvey)
```

---

Learning EdSurvey
========================================================

- Reading vignettes provided in training materials

```r
vignette("introduction", package="EdSurvey")
```

- R help

```r
help(package = "EdSurvey")
```

- [EdSurvey eBook](https://naep-research.airprojects.org/Portals/0/EdSurvey_A_Users_Guide/_book/index.html)

- [EdSurvey Website](https://www.air.org/project/nces-data-r-project-edsurvey)

- [EdSurvey Github](https://github.com/American-Institutes-for-Research/EdSurvey)

- [NAEP Data Training workshop](http://naep-research.airprojects.org/Opportunities/Training)

---

Self-Reflection - R Functions
========================================================

**Ask yourself:** What are the arguments of the function `readNAEP()`? What are some examples of acceptable values for each argument?

---

Self-Reflection - R Functions
========================================================

`readNAEP()` arguments, from R documentation (*type `?readNAEP` in the console*)

![](data:image/png;base64,#presentationFigures/readNAEP.png)

---

Data Processing
========================================================
<hr class="sectionSlide">
.logoSection[
<img src="data:image/png;base64,#presentationFigures/naepLogo.png" width="80%" height="80%" />
]

---

Data Processing
========================================================

- First, read in the publicly available NAEP data from `NAEPprimer`

```r
sdf <- readNAEP(system.file("extdata/data", "M36NT2PM.dat", package = "NAEPprimer"))
ita <- readTIMSS("~/TIMSS/2019/", countries="ita", grade=8)
```

```
## Found cached Grade 8 data for country code "ita: Italy".
## edsurvey.data.frame data level detail:
## |---DataLevel----|----Rows----|--Columns---|---MergeType----|-------MatchedRecords-------|-OK-|
## |Student         |        3619|        3196|                |*base level*                | ✓  |
## |>School         |        3619|          89|many:one        |3619 of 3619                | ✓  |
## |>Teacher        |        7238|         766|one:many        |7238 of 7238                | ✓  |
```

- How long did that take?
- Try it again, then how long?

---

Data Processing
========================================================

- Can also read in other restricted-use NAEP data by naming the file location

```r
math17 <- readNAEP("//path_to_directory/Data/M48NT2AT.dat")
```

- The first character indicates the subject - `M` (Math)
- The second and third characters indicate the NAEP year - `48` (2017 - 1969 = 48)
- The fourth character indicates the component - `N` (National)
- The fifth character indicates the type of data - `T` (Student)
- The sixth character indicates the grade cohort - `2` (8th)
- The seventh and eighth characters indicate the sample - `AT` (Main NAEP)

---

Data Processing
========================================================

NOTE: the dat file requires its intact data folder directory in order to be read in correctly; containing both the student and school level files to merge data

---

Meet Your Data
========================================================
<hr class="sectionSlide">
.logoSection[
<img src="data:image/png;base64,#presentationFigures/naepLogo.png" width="80%" height="80%" />
]

---

Quick Terminology Notes
========================================================

The `edsurvey.data.frame` class stores information about survey data via a data connection, which allows for:
  - Correct calculation of relevant statistics.
  - Limited working memory usage.

<br>

**`_____SDF`**

The Edsurvey package uses the acronym `SDF` in the names of several functions to signify their relationship to **S**urvey **D**ata **F**rames.

---

Quick Terminology Notes
========================================================

The `edsurvey.data.frame.list` class stores a list of `edsurvey.data.frame` objects.

- The list can be passed to the analysis functions, and a result list will be returned.
  - A list can store both `cov` (covariants) and `labels` arguments. For example, the 'year' and 'country' might vary across the `edsurvey.data.frame`s in the list.
  - `edsurvey.data.frame.lists` can also be constructed manually.  See the `?edsurvey.data.frame.list` documentation.

---